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ABSTRACT 


Naval  Air  Systems  Command’s  (NAVAIRs)  weapons  test  ranges  at  China  Lake, 
CA  struggle  to  meet  increasing  demand.  Development  programs  are  bringing  more 
complex  and  capable  weapons  to  the  ranges.  The  resources  of  the  ranges  are  being 
stretched  thin  and  not  all  requests  for  testing  are  accommodated.  The  purpose  of  this 
paper  is  to  seek  a  solution  to  increase  range  throughput  within  the  constraints  of  the 
current  resources. 

The  effort  involved  evaluating  range  usage,  identifying  obstacles  to  increased 
throughput,  and  evaluating  the  processes  associated  with  the  obstacles. 
Recommendations  for  process  changes  were  made  and  applied  to  a  set  of  historical  data 
to  detennine  the  impact  of  the  processes  and  compare  them  with  the  historical  solution. 
Data  from  the  analysis  show  that  specific  changes  to  current  processes  have  the  potential 
to  increase  throughput  by  9%  without  the  need  for  additional  resources. 
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EXECUTIVE  SUMMARY 


NAVAIR’s  ranges  are  national  assets.  The  investment  in  infrastructure, 
instrumentation,  and  other  resources  is  significant  and  the  amount  of  land  set  aside  for 
testing  cannot  be  duplicated.  Department  of  Defense  weapons  acquisition  programs  are 
dependent  on  ranges  to  perform  testing  in  support  of  acquisition  decisions.  It  is  critical 
that  ranges  be  managed  to  provide  the  greatest  possible  throughput  to  meet  increasing 
demands. 

This  study  at  NAVAIR’s  China  Lake  ranges  demonstrates  how  additional 
capacity  can  result  from  applying  systems  engineering  principles.  Analysis  indicates 
processes  can  be  changed  to  encourage  less  frequent  movement  of  instrumentation  assets. 
Fewer  moves  equates  to  less  setup  time,  less  wear  and  tear  on  instrumentation  systems, 
and  more  time  for  testing.  Choosing  to  accept  the  recommendations  of  this  paper  will 
provide  new  focus  on  meeting  increasing  customer  requirements  by  working  more 
efficiently  with  the  resources  currently  available.  Initial  analysis  indicates  that  an 
increase  of  9%  in  available  range  time  can  be  obtained  at  no  cost  by  using  these 
rec  ommendations . 
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I.  INTRODUCTION 


A.  BACKGROUND 

Open-air  weapons  ranges  were  established  at  China  Lake,  California,  as  part  of 
the  establishment  of  the  Naval  Ordnance  Test  Station  (NOTS)  in  1943  to  support  Test 
and  Evaluation  (T&E)  of  rockets  and  aviation  ordnance.  For  the  past  65  years  these 
ranges  have  supported  weapons  Research  and  Development  (R&D)  programs  for  the 
Navy  as  well  as  other  US  defense  services,  defense  agencies,  allied  forces,  and  private 
industry.  Access  to  instrumented  test  ranges  with  appropriate  infrastructure,  resources, 
and  availability  is  essential  to  obtaining  “decision  quality  data”  in  support  of 
programmatic  decisions. 

The  China  Lake  ranges  are  managed  by  the  Range  Department  of  the  Naval  Air 
Systems  Command  (NAVAIR)  and  comprise  over  1,700  square  miles  of  land  and  20,000 
square  miles  of  controlled  airspace.  The  ranges  carry  out  testing  on  a  wide  variety  of 
systems  and  components  across  the  entire  spectrum  of  military  systems.  The  increasing 
complexity  of  weapons  systems  brings  a  need  for  an  equally  complex  test  environment. 
There  is  a  continuing  trend  to  increase  the  distance  weapons  are  able  to  travel  after 
launch  or  release.  Testing  such  systems  requires  larger  portions  of  the  range,  or  even  the 
combined  resources  of  several  ranges,  to  perform  adequate  tests.  These  range  challenges 
are  compounded  by  years  of  forced  manpower  downsizing  and  reduced  budgets. 
Although  the  ranges  conduct  hundreds  of  test  operations  per  month,  they  are  not  able  to 
accommodate  ah  requests  for  testing  with  existing  resources  using  the  current  processes. 

B.  PURPOSE 

The  purpose  of  this  thesis  is  to  analyze  range  processes,  identify  factors  that 
restrict  throughput,  and  propose  modifications  to  processes  that  will  result  in  additional 
available  time  for  testing. 
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C.  RESEARCH  QUESTION 

Can  the  range  throughput  be  increased  by  analyzing  existing  processes  and 
applying  system  engineering  principles  to  modify  those  processes? 

D.  BENEFITS  OF  STUDY 

The  thesis  will  provide  recommendations  for  process  changes  to  improve 
utilization  of  range  time  and  other  range  resources.  Impact  of  suggested  changes  on 
shareholders  and  on  other  processes  will  be  identified  to  provide  decision  makers  all  the 
data  needed  for  a  decision  on  implementing  the  changes.  The  range  will  benefit  from  the 
study  by  being  presented  options  for  improved  utilization. 

E.  SCOPE  AND  METHODOLOGY 

This  thesis  focuses  on  the  China  Lake  Range  complex  of  NAV AIR’s  Ranges 
Department.  While  policy  and  resource  issues  are  increasingly  common  with  the 
Electronic  Combat  Range  on  the  southern  portions  of  the  same  station,  there  is  no  attempt 
to  include  Electronic  Warfare  (EW)  testing  in  this  analysis.  The  missions,  test  resources, 
and  scenario  designs  are  different  enough  that  EW  throughput  issues  should  be  addressed 
independently. 

Analysis  was  perfonned  on  processes,  historical  scheduling  data,  and  on 
information  provided  in  interviews  with  personnel  of  the  range  department  and  with 
range  customers.  The  scope  of  policies  and  processes  analyzed  is  limited  to  those  under 
control  of  the  range  department  (NAV AIR  5.2).  Increasing  hours  of  operations, 
manpower,  and  number  of  range  support  systems  is  outside  the  scope  of  this  study. 

The  range  is  committed  to  getting  as  many  programs  on  the  schedule  as  possible, 
but  program  priorities  must  be  taken  into  account.  It  is  not  within  the  scope  of  this  study 
to  disregard  relative  program  priority  or  to  seek  improvement  by  excluding  part  of  the 
current  customer  base  in  favor  of  another. 
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II.  BACKGROUND  RESEARCH 


A.  INTRODUCTION 

The  information  in  this  chapter  is  intended  to  present  enough  detail  about  range 
operations  and  range  processes  to  provide  a  foundation  for  understanding  the  information 
presented  in  the  rest  of  the  paper.  The  range  is  a  large  and  complex  system  with  many 
interrelated  components  and  significant  external  inputs.  There  is  no  attempt  to  put  forth 
complete  details  on  any  portion  of  the  range  system.  Emphasis  is  placed  on  the  internal 
processes  and  interactions  that  are  within  the  influence  of  the  Range  Department  that  will 
receive  the  recommendations  contained  in  this  thesis. 

B.  CUSTOMER  REQUIREMENTS  PROCESS 

The  Test  Management  Branch,  Code  52130MD,  is  the  designated  entry  point  for 
customers  who  need  to  test  on  the  ranges.  Customers  are  directed  to  Test  Management 
by  range  literature,  presentations,  web-sites,  and  by  word  of  mouth.  The  test  manager 
assigned  to  a  program  is  responsible  for  learning  the  customers’  requirements  well 
enough  to  determine  the  feasibility  of  conducting  testing  on  the  ranges.  Once  feasibility 
has  been  established  the  test  manager  works  with  all  involved  parties  to  use  the  test 
requirements  of  the  program  to  design  specific  test  events.  These  test  events  are  designed 
and  conducted  so  that  the  resulting  data  meet  the  requirements  for  customers  to  evaluate 
the  performance  of  their  item  under  test. 

Acquisition  programs  following  the  standard  cycle  will  have  several  phases  of 
testing.  Developmental  Testing  (DT)  is  under  the  control  of  the  responsible  organization. 
This  organization  may  be  within  the  government  or  it  may  be  a  contractor. 
Developmental  test  programs  have  a  high  degree  of  requirements  variability  from  test  to 
test  as  they  verify  design  concepts  and  cycle  through  test-fix-test.  After  the  program 
enters  Operational  Test  (OT)  there  is  a  much  more  fonnal  process  for  the  testing  and  for 
validation  of  test  requirements.  In  OT  the  responsible  test  organization,  such  as 
Commander  Operational  Test  and  Evaluation  Force  (COMOPTEVFOR)  for  the  Navy, 
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will  have  written  a  Test  and  Evaluation  Master  Plan  (TEMP)  that  clearly  defines  key 
performance  parameters.  This  plan  is  signed  by  the  tester  but  also  by  the  services’ 
executive  agent  for  testing.  The  executive  agent  for  Navy  testing  is  N091.  A  program 
TEMP  will  define  test  objectives  and  the  plan  and  schedule  for  meeting  the  objectives. 
The  program’s  test  engineer  works  with  the  ranges  to  determine  the  details  of  the  test 
design  and  how  range  resources  will  support  the  test. 

Some  customers  have  extensive  experience  on  test  ranges  and  bring  well 
developed  test  plans  with  clear  requirements  tied  to  specific  test  objectives.  New 
programs,  or  those  with  less  experienced  test  teams,  may  need  coaching  and  interaction 
with  the  test  manager  and  other  range  support  personnel  experienced  in  test  design. 
Requirements  established  during  early  customer  contact  become  design  constraints,  as 
they  drive  all  aspects  of  test  scenario  design  and  conduct.  The  event  plans  must  also 
ensure  tests  are  conducted  within  the  bounds  of  range  policy  with  regards  to  safe  conduct 
and  regulatory  compliance. 

Test  managers  are  guided  by  a  Test  Managers  Processes  handbook  (Test 
Management  Processes,  2007).  The  book  is  a  collection  of  instructions,  guidance,  policy, 
and  procedures.  New  documents  and  updates  to  existing  documents  are  added  as  needed. 
Within  the  handbook  are  flow  charts,  forms,  and  checklists  that  guide  the  requirements 
collection  process. 

Test  event  design  is  a  product  of  customer  requirements  and  considerations 
required  for  safe  test  conduct.  Customer  test  parameters  drive  decisions  on 
instrumentation  requirements.  Factors  that  drive  test  design  decisions  detennine  test 
constraints.  Examples  of  program  requirements  that  drive  test  design  are: 

•  Test  objectives,  including  measurement  details  such  as  units,  accuracy, 
and  resolution 

•  Key  performance  parameters 

•  Other  details  contained  in  program  documentation  such  as  a  TEMP 
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C.  CURRENT  PROCESS  FOR  DETERMINING  RANGE  RESOURCE 
REQUIREMENTS 

Establishing  range  resource  requirements  for  an  event  requires  the  interaction  of  a 
team  of  experienced  resource  providers.  The  Range  Department  has  several  divisions 
which  provide  resources  in  support  of  testing.  Each  division  has  staff  to  assist  in 
assessing  proper  test  design  and  test  conduct.  Resources  provided  or  controlled  within 
the  range  divisions  include: 

•  Air  operations  support  (Airspace  Controllers) 

•  Airspace  surveillance 

•  Range  access  control 

•  Ground  test  conductors 

•  Range  land  space 

•  Range  airspace 

•  Communications  support 

o  Radios 
o  Land  lines 

o  Access  to  range  and  off-range  digital  network  for  video,  audio, 
and  other  sensor  data  transport 

•  Computer  operations 

•  Fixed  video:  6  options 

•  Tracking  cameras:  4  options 

•  Frequency  spectrum  monitoring 

•  Generators 

•  Global  Positioning  System  (GPS)  air  and  ground  systems 

•  Radar  tracking  and  radar  directed  video  support:  1 5  options 

•  Ground  laser  and  spot  video  support 

•  Telemetry  systems:  17  options 

•  Range  cleanup 

•  Scoring  support 

•  Ground-based  stationary  targets 
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•  Air  vehicle  instrumentation 

•  Weapons  instrumentation 

•  Weather  data  (pre-test  and  real-time  if  needed) 

Additional  resources  outside  the  range  department  that  require  coordination 
include: 

•  Moving  and  airborne  targets 

•  Aircraft 

•  Airfield  access 

•  Ordnance  handling  and  tracking 

A  detailed  test  design  will  detennine  the  number  and  mix  of  resources  required. 
An  example  is  the  requirement  for  providing  Time  Space  Position  Information  (TSPI)  of 
an  item  dropped  from  an  aircraft.  Both  radar  (Figure  1)  and  Kineto  Tracking  Mounts 
(KTMs)  (Figure  2)  can  be  used  to  obtain  TSPI  depending  on  the  required  accuracy.  The 
radar  solution  uses  one  or  more  of  several  radar  systems  at  fixed  locations  on  the  range  to 
obtain  TSPI.  Setup  time  for  the  radar  is  less  than  30  minutes.  The  KTMs  can  collect 
calibrated  video  data  for  processing  that  will  yield  higher  precision  TSPI,  but  with  added 
cost  in  resources  and  time.  KTMs  are  mobile  systems  that  are  transported  to  the  test 
location  and  then  require  a  two-hour  calibration  time.  The  processing  of  the  KTM  data  to 
yield  TSPI  requires  manual  processing  of  each  frame  of  image  data  and  uses  triangulation 
to  detennine  precise  location.  Processing  takes  hours  for  each  minute  of  test  data.  Since 
triangulation  is  used,  a  TSPI  solution  requires  multiple  KTMs  with  complimentary  views 
of  the  test  item.  Once  again,  the  difference  between  a  simple  radar  solution  and  a 
complex  KTM  solution  is  traced  back  to  required  resolution  accuracy.  An  overstatement 
of  requirements  can  result  in  unneeded  setup  time,  expensive  data  products,  and 
significantly  longer  time  to  provide  a  data  product. 
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Figure  1.  This  remotely  operated  tracking  radar  is  an  example  of  an  instrumentation 
resource  that  is  at  a  fixed  location  and  cannot  be  moved. 


Figure  2.  One  of  15  Kineto  Tracking  Mounts  (KTMs)  with  instrumentation  cameras. 

KTM  systems  are  mounted  on  trailers  and  towed  from  one  location  to  another  as 

needed. 

Determination  of  required  number  and  location  of  KTMs  is  itself  an  engineering 
challenge.  The  range  supports  12  mobile  KTMs  that  can  be  located  at  any  combination 
of  over  200  surveyed  KTM  pads  on  the  range.  Geometry  of  the  planned  target  path  with 
relation  to  the  other  KTM  locations  determines  achievable  accuracy.  An  experienced 
instrumentation  team  determines  the  required  setup  based  on  the  requirements  presented 
by  the  test  manager. 

Fixed  video  systems  are  often  deployed  in  close  proximity  to  ground  targets  to 
provide  documentation  of  weapon  impact.  These  systems  are  remotely  operated  because 
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they  are  inside  the  hazard  areas.  The  requirement  for  “still”  video  instrumentation 
requires  precise  placement  and  positioning  of  remotely  operated  cameras  at  individually 
surveyed  locations  as  shown  in  Figure  3.  Customer  requirements  drive  the  number  of 
cameras  and  geometry  of  the  camera  placement.  When  the  mission  is  to  record  a  weapon 
engagement  with  a  moving  target,  the  camera  setup  must  include  the  entire  area  of  the 
motion  of  the  target  during  the  critical  portions  of  the  test  event.  This  added  complexity 
can  require  up  to  20  cameras  and  increase  the  setup  time  to  an  entire  workday. 


Figure  3.  Fixed  video  camera  and  support  equipment  placed  to  instrument  a  test. 

Some  laser  support  capability  exists  within  the  range’s  Instrumentation  Division, 
but  most  tests  require  support  from  outside  the  Range  Department.  The  setup  for  laser 
support  is  also  complex  and  time  consuming.  The  lookdown  angle  of  the  required 
equipment  requires  camera  placement  on  towers  as  shown  in  Figure  4.  The  specifics  of 
the  test  will  detennine  which  type  of  tower  can  be  used.  Nearly  all  applications  rely  on 
mobile  towers  that  must  be  transported  and  erected  before  they  can  be  used. 
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Figure  4.  Laser  support  van  and  two  towers  used  to  support  laser  scoring 
instrumentation.  Note  the  mobile  tower  and  base. 

There  are  limited  numbers  of  GPS,  telemetry,  and  radar  systems  at  the  range. 
Most  of  these  resources  are  at  fixed  locations.  Requirements  for  these  resources  are 
determined  by  required  bandwidth,  frequency  de-confliction,  and  the  geometry  and 
ground  topology  to  ensure  line  of  sight  between  the  resource  and  the  item  under  test. 
These  resources  must  be  scheduled  based  on  test  priority.  For  many  of  these  systems  it  is 
not  possible  to  increase  capacity  by  purchasing  additional  systems  as  the  availability  and 
allocation  of  frequency  spectrum  is  a  limiting  factor. 

Chapter  IV  will  further  address  which  of  these  resources  are  key  constraints  and 
how  that  is  determined. 

D.  CURRENT  SCHEDULING  PROCESS 

All  events  conducted  on  the  Land  Ranges  are  scheduled  through  the  Land  Range 
Test  Management  Branch  Scheduling  Office.  The  schedule  of  range  test  events  for  a 
given  week  is  locked  in  and  published  on  Wednesday  of  the  prior  week.  Before  being 
considered  for  a  slot  on  the  range  schedule,  the  detailed  event  plan  must  be  approved  by 
multiple  range  support  offices  including:  finance,  test  management,  test  operations,  range 
data  systems,  frequency  management,  laser  safety,  range  safety,  the  environmental  office, 
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and  explosive  ordnance.  Inputs  to  the  scheduling  process  are  in  the  form  of  electronic 
submissions  using  a  local  tool  (the  Test  Resource  Management  System  (TRMS))  and 
include: 

•  Requested  date  and  time  for  the  event  with  alternative  dates  and  times,  if 
any 

•  Aircraft  schedule  from  the  squadrons  supporting  testing  (some  come  with 
priority  rankings) 

•  Event  plan  containing 

o  Target  description  and  location 
o  Range  area  required  for  conduct 
o  Test  bay  requirements 
o  Telemetry  requirements 
o  Radar  support  requirements 

o  Requirements  for  still  video,  tracking  video,  and  laser  setup 
o  Weapons  to  be  used 
o  Use  of  laser,  jamming,  or  other  emissions 

o  Map  with  event  data  such  as  captive  carry  routes,  run-in  headings, 
release  point,  target  locations,  and  safety  footprints 

o  Required  evacuation  and  road  closure  details 

Each  Wednesday  the  scheduling  office  personnel  take  the  inputs  provided  above 

and  begin  the  process  of  building  the  next  week’s  schedule.  Day  by  day  they  consider 

the  requested  tests  and  attempt  to  fit  as  many  as  possible  onto  the  schedule.  There  are 

priority  considerations  provided  by  the  local  test  wing  as  well  as  by  the  test  management 

branch  head  who  has  the  final  say  on  scheduling. 

The  variability  associated  with  a  wide  variety  of  test  complexity  adds  significant 
challenges  to  the  scheduling  process.  Video  instrumentation  setup  time  is  an  hour  for 
some  tests  and  6  to  8  hours  for  other  tests.  Instrumentation  crews  can  work  safely  under 
some  ongoing  tests  in  preparation  for  the  next  test,  but  must  evacuate  to  safe  locations 
during  others.  Setup  time  for  test  bays  can  be  30  minutes  or  60  minutes  depending  on 
test  specifics.  Some  tests  can  run  concurrently  on  different  portions  of  the  range  if 
adequate  supporting  resources  are  available.  At  times  the  same  portion  of  the  range  can 
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be  used  by  two  testers  with  appropriate  altitude  buffers.  The  scheduling  office  spends  a 
great  deal  of  time  on  scheduling  day  on  the  phone  with  test  managers  and  various 
resource  providers  to  find  areas  where  constraints  can  be  relaxed  to  allow  better  range 
utilization  and  to  insure  all  tests  scheduled  have  the  required  resources  and  approvals. 

The  schedulers  work  to  have  a  viable  draft  schedule  approved  by  the  branch  head 
and  ready  for  the  weekly  scheduling  meeting  that  takes  place  at  2:30  pm  every 
Wednesday.  Frequency  management,  test  controllers,  air  controllers,  test  managers,  and 
resource  providers  attend  the  scheduling  meeting.  The  schedule  is  discussed  in  detail 
with  opportunities  for  input  from  participants  on  issues  that  may  still  need  further 
evaluation  to  avoid  conflicts.  It  is  common  for  one  or  more  events  to  have  changes  made 
in  the  meeting  or  shortly  thereafter  based  on  inputs  from  the  meeting.  After  the  meeting 
the  schedule  is  updated  to  reflect  necessary  changes.  The  schedule  is  reviewed  and 
approved  by  the  branch  head  a  final  time  and  the  schedule  is  locked  in  no  later  than  9:00 
am  on  Thursday. 

E.  CHAPTER  SUMMARY 

Information  has  been  presented  to  demonstrate  the  complexity  of  the  range  and 
provide  a  frame  of  reference  for  the  analysis  that  will  be  presented.  The  validation  of  test 
objectives  and  test  setup  has  been  discussed.  There  are  complex  interactions  among  the 
various  parts  of  the  range  and  there  are  real  test  consequences  in  cost  and  schedule 
impact  for  choices  made  along  the  way. 


11 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


12 


III.  APPLICATION  OF  STUDY 


A.  INTRODUCTION 

In  this  chapter  the  analysis  of  the  range  system  will  begin  with  a  discussion  of 
interfaces  and  decision  points.  The  role  of  a  range  software  tool  in  the  process  will  be  put 
in  context,  and  historic  test  data  from  a  four  week  period  will  be  selected  as  a  baseline 
against  which  proposed  changes  will  be  measured.  Lastly  the  measure  of  effectiveness 
for  assessing  results  will  be  defined. 

B.  PROCESS  INTERFACES  AND  DECISION  POINTS 

There  are  many  interfaces  and  decision  points  in  the  test  planning  and  execution 
processes.  It  is  convenient  to  look  at  the  interfaces  and  decision  points  chronologically  in 
five  phases. 

1.  Phase  1  -  Early  Customer  Contact 

The  first  interface  is  between  the  potential  customer  and  a  test  manager.  In  this 
interface,  the  customer  provides  enough  detail  about  the  program’s  test  objectives  and 
required  conditions  to  allow  the  test  manager  to  determine  the  feasibility  of  conducting 
the  test  on  the  range.  If  a  program  requests  conditions  or  resources  beyond  typical  range 
capability,  further  discussions  will  need  to  occur  between  the  customer  and  range 
engineering  staff  to  determine  if  the  requested  test  can  be  accommodated. 

2.  Phase  2  -  Event  Planning 

Event  planning,  the  most  time  consuming  part  of  the  process,  is  the  second  phase 
and  involves  the  most  decision  points.  In  this  phase,  the  test  manager  uses  the  customer’s 
test  plan  as  a  starting  point  to  make  a  series  of  decisions  that  results  in  a  detailed  event 
plan.  Each  event  plan  supports  a  test  scenario,  or  set  of  scenarios,  that  measure  the  item’s 
performance  against  the  documented  program  objectives.  Decisions  are  made  on  upper 
level  requirements  for  instrumentation  support.  The  interface  at  this  level  is  typically 
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between  the  test  manager  and  the  customer.  Upper  level  decisions  would  include  items 
such  as  the  need  for  TSPI  data  and  the  associated  accuracy,  video  documentation 
showing  a  particular  aspect  of  the  test  item  or  delivery  vehicle,  or  a  telemetry  link  and 
display  to  show  weapons  status  before  and  after  launch. 

Within  this  same  event  planning  phase  decisions  must  be  made  at  the  next  level  of 
complexity.  These  decisions  define  the  specific  test  setup  to  meet  requirements  decided 
on  at  the  upper  level.  An  example  would  be  the  exact  number  and  placement  of  cameras 
to  capture  the  image  of  the  test  item  at  a  precise  time  in  the  test  event.  This  includes 
flight  path,  release  time,  and  release  condition  decisions.  The  interfaces  at  this  decision 
step  include  test  management  and  all  involved  resource  providers  as  well  as  the  customer 
when  needed. 

The  conclusion  of  the  event  planning  phase  is  achieved  when  all  approvals 
required  to  achieve  “ready  to  conduct”  status  have  been  obtained.  Each  approval  is  a 
decision  point.  Only  after  all  approvals  have  been  obtained  is  a  test  considered  for 
scheduling. 


3.  Phase  3  -  Test  Scheduling 

Decisions  in  this  phase  are  date,  time,  and  location  of  test  event.  Supporting 
aircraft  and  each  range  resource  to  be  used  are  also  determined.  Interfaces  are  through 
test  managers  to  the  customers  and  through  test  managers  and  the  schedulers  to  each 
supporting  resource  provider.  Final  decision  for  scheduling  of  test  events  resides  with  the 
head  of  the  test  management  branch. 

4.  Phase  4  -  Test  Conduct 

In  the  test  conduct  phase,  the  test  conductor  from  the  Range  Operations  Division 
conducts  the  test  event  according  to  the  event  plan.  The  test  conductor  is  the  decision 
maker  during  an  event.  Nearly  all  airborne  events  are  conducted  from  one  or  more  test 
bays  in  the  Range  Control  Center.  The  test  bay  is  set  up  with  displays  from  the  various 
supporting  instrumentation  resources  which  allow  the  test  conductor  and  customer  real¬ 
time  feedback  on  the  item  under  test,  the  supporting  resources,  and  the  test  environment. 
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Customers  work  with  the  test  conductor  to  work  through  the  planned  event.  If  portions 
need  to  be  repeated,  skipped,  or  modified  within  the  approved  parameters  of  the  event,  it 
is  the  test  conductor  that  makes  that  decision.  All  participants  have  the  ability  to 
tenninate  the  event  for  reasons  of  safety  if  they  perceive  an  unsafe  situation  exists. 
Interfaces  in  this  phase  include  verbal  communications  within  the  bay,  electronic  voice 
communications  within  the  bay  and  with  test  participants  across  the  range,  and 
communications  with  air  controllers  at  the  range  and  within  the  larger  test  complex. 

5.  Phase  5  -  Data  Production  and  Delivery 

The  required  data  product  defined  by  the  customer  in  planning  and  documented  in 
the  event  plan  determines  what  data  are  collected  and  how  they  are  processed  during  and 
after  the  test.  All  customer  deliverable  products  pass  through  a  quality  check  to  ensure 
correct  processing.  While  many  post-test  data  products  are  specified  prior  to  the  test  as 
part  of  the  event  plan,  it  is  possible  for  some  additional  products  to  be  requested  if  the 
instrumentation  and  collected  data  support  that  product.  All  requests  for  test  data 
products  are  made  by  customers  through  the  test  manager.  Interfaces  include  the 
customer,  test  manager,  and  personnel  in  the  Range  Data  Systems  Division. 

The  infonnation  on  interfaces  and  decision  points  is  summarized  in  Table  1. 
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Phase 

Interfaces 

Decision  Points 

1 .  Early  Customer 
Contact 

Customer  and  Test  Manager 

Feasibility  of  test 

2.  Event  Planning 

Customer  and  Test  Manager 

Test  Manager,  Resource 
Providers,  and  Customer 

Test  Manager,  Test  Approvers 

Event  Plan  Approval  by: 

Test  Management 

Test  Operations 

Range  Data  Systems 

Frequency  Management 

Laser  Safety 

Range  Safety 

Environmental  Office 

Explosive  Ordnance 

3.  Test  Scheduling 

Scheduling  Office,  Test 

Manager,  Customers,  Resource 
Providers 

Scheduled  Event  (day,  time, 
location,  test  conditions) 

4.  Test  Conduct 

Test  Conductor,  Customers, 
Resource  Provider,  Test 

Manager 

Go-no-go 

Test  conduct  decisions 

5.  Data  Production 
and  Delivery 

Data  Systems  Personnel, 
Customer,  Test  Manager 

Correct  and  sufficient  data 

Table  1 .  Summary  of  interfaces  and  decision  points  in  the  life  cycle  of  a  test  program. 


6.  Role  of  TRMS  in  Interfaces  and  Decision  Tracking 

Since  its  development,  the  TRMS  system  has  become  a  living  repository  for  the 
details  of  the  event  plan.  TRMS  is  accessible  by  all  test  resource  providers  and  is 
intended  to  be  used  as  an  interface  to  distribute  and  coordinate  details  of  test  plans. 
TRMS  does  not  replace  the  face-to-face  interactions  that  form  the  primary  interfaces 
between  test  participants,  but  it  does  serve  as  an  automated  tool  for  soliciting  and 
recording  decisions  in  the  fonn  of  approvals  from  the  various  individuals  with  range 
decision  authority  prior  to  the  conduct  of  a  test. 
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C.  SELECTION  OF  REPRESENTATIVE  TEST  PERIOD  FROM 

HISTORICAL  DATA 

Historical  data  archived  in  the  scheduling  office  provides  much  of  the  data  needed 
for  analysis  required  in  this  report.  There  is  variability  in  workload  at  the  ranges.  In 
periods  of  lighter  workload  less  annotation  exists  in  the  records  and  time  spacing  between 
events  is  more  difficult  to  attribute  to  a  specific  cause.  The  scheduling  records  alone  do 
not  provide  a  complete  picture.  For  example  they  do  not  show  where  extensive  amounts 
of  time  were  spent  setting  up  instrumentation  outside  normal  working  hours.  Despite  the 
lack  of  such  data,  the  scheduling  records  still  give  the  most  complete  view  available  from 
any  single  source. 

Criterion  for  selection  was  a  period  of  high  workload  where  there  was  more 
demand  for  range  time  and  resources  than  could  be  provided.  This  ensures  there  was  a 
good  reason  for  range  periods  that  were  not  used  for  test  conduct  and  increases  the 
likelihood  of  detailed  annotation  in  the  scheduling  records.  While  the  range  workload  is 
variable,  there  is  also  an  element  of  periodicity.  Due  to  the  nature  of  funding  cycles  there 
is  a  fiscal  year  impact  that  has  historically  resulted  in  high  demand  for  testing  in  the  last 
few  months  before  the  end  of  the  fiscal  year.  A  review  of  the  scheduling  records  for  July, 
August,  and  September  2007  indicates  August  had  sufficient  demand  as  well  as  adequate 
annotation  for  analysis  purposes.  The  period  of  August  1  to  18  was  chosen  as  the  time 
period  for  a  baseline  against  which  to  test  the  impact  of  proposed  changes. 

D.  DETERMINE  MEASURES  OF  EFFECTIVENESS 

The  Measure  of  Effectiveness  (MOE)  for  this  study  is  the  difference  in  number  of 
range  hours  available  for  testing.  In  this  context,  the  MOE  is  only  applicable  looking 
back  at  historical  scheduling  data  and  comparing  them  with  the  results  of  re-scheduling 
after  applying  proposed  process  changes  to  the  same  set  of  testing  requests. 
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E. 


CHAPTER  SUMMARY 


Chapter  III  identified  interfaces  and  decision  points  critical  to  relevant  processes 
that  will  be  used  in  the  next  phase  of  analysis.  A  period  of  time  was  chosen  where 
sufficient  historical  data  existed  to  be  used  as  a  baseline  for  future  comparison,  and  an 
MOE  was  selected  to  be  used  in  quantifying  the  results  of  proposed  changes.  The 
groundwork  is  now  set  to  focus  analysis  efforts  on  the  problem  of  reducing  restricted 
range  time. 
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IV.  RESEARCH  ANALYSIS 


A.  INTRODUCTION 

The  object  of  this  chapter  is  to  describe  the  process  used  in  selecting  which  range 
processes  were  analyzed,  which  ones  merited  proposed  changes,  and  what  the  results  of 
the  changes  were  when  applied  to  the  baseline  period  selected.  Historical  scheduling 
data  are  used  to  find  potential  causes  of  restricted  range  time.  After  the  causes  are 
identified,  the  processes  associated  with  the  causes  are  reviewed  and  evaluated  for 
possible  changes  to  reduce  their  contribution  to  restricted  range  periods.  The  suggested 
process  changes  are  applied  retroactively  to  a  baseline  historical  period  and  the  MOE  is 
used  to  detennine  the  effect  of  the  changes. 

B.  IDENTIFY  MAJOR  CAUSES  OF  INEFFICIENT  RANGE  USAGE 

At  this  stage  certain  assumptions  are  made  to  focus  the  analysis  on  data  with  the 
highest  potential  for  yielding  additional  usable  range  time.  The  first  assumption  is  that 
the  amount  of  time  scheduled  for  each  test  is  appropriate.  The  ability  to  carry  out  all  the 
activities  required  in  an  event  plan  within  an  acceptable  amount  of  risk  to  the  program  is 
best  determined  by  the  test  engineer  and  test  manager.  Test  time  scheduled  is  not 
analyzed  for  efficiency. 

A  second  assumption  is  that  activities  supporting  test  events  take  precedence  over 
non-test-support  activities.  As  a  result,  non-test-support  activities  do  not  generally 
compete  with  test  events  for  time  on  the  schedule.  Examples  of  non-test-support 
activities  include  support  of  range  infrastructure  such  as  power  distribution,  and 
environmental  stewardship  activities  like  studies  of  geology,  flora,  fauna,  insects,  and 
animals.  Since  these  activities  are  scheduled  on  a  not-to-interfere  basis,  they  are  not 
analyzed  for  impact  to  range  throughput. 

The  final  assumption  is  that  the  setup  time  required  to  prepare  a  resource  for  any 
given  test  is  valid  and  cannot  be  reduced.  No  attempt  will  be  made  in  this  study  to 
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suggest  that  resource  providers  need  less  time  to  setup  their  respective  systems  in  support 
of  a  test.  In  light  of  the  stated  assumptions,  the  activities  that  are  left  for  analysis  are 
those  that  directly  support  test  events  and  require  scheduling  consideration,  but  are  not 
performed  during  the  actual  scheduled  test  event  time.  These  activities  include  the 
range’s  preparation  for  testing  such  as  the  setup  of  targets,  instrumentation,  and  other 
activities  needed  for  successful  test  conduct. 

Scheduling  records  show  every  event  that  was  scheduled  as  well  as  those 
requested  that  did  not  make  the  schedule,  with  annotation  for  the  reason  each  was  not 
scheduled.  These  records  also  show  details  such  as  test  location  and  resources  that  were 
used  in  support  of  the  event.  Setup  time  can  be  determined  from  the  location  and  list  of 
associated  resources. 

Twelve  weeks  of  historical  data  were  used  as  the  basis  for  analysis.  These  weeks 
were  contiguous  weeks  within  July,  August,  and  September  2007.  This  timeframe 
includes  the  period  chosen  as  the  baseline  for  evaluation  for  the  effectiveness  of  proposed 
changes. 

From  the  resources  listed  previously,  four  were  identified  that  require  range  setup 
time.  They  are  KTM  systems,  laser  systems,  target  preparation,  and  video  setup.  It  was 
found  that  the  construction  and  setup  of  targets  on  the  range  are  performed  around  other 
test  events  and  are  usually  completed  well  in  advance  of  the  required  time.  As  such, 
target  construction  and  setup  do  not  require  scheduling  consideration. 

The  12-week  period  analyzed  contained  365  test  events.  Data  for  further  analysis 
were  collected  only  on  the  97  events  that  required  one  or  more  of  the  three  targeted 
resources.  The  hours  required  for  setup  of  each  resources  was  determined  by  the 
resource  provider.  The  data  considered  are  test  events  with  associated  setup  time  for 
KTM,  video,  or  laser  resources.  The  impact  of  setup  time  on  the  range  schedule  is  the 
longest  time  required  by  any  one  of  the  required  resources.  The  detailed  data  for  the 
first  week  is  presented  in  Table  2. 
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Date 

Test  # 

KTM  Setup 
Time 

Video  Setup 
Time 

Laser  Setup 
Time 

Range  Hours 
Impacted 

7/3/2007 

1 

3 

8 

0 

8 

7/3/2007 

2 

0 

5 

0 

5 

7/5/2007 

3 

0 

0 

4.5 

4.5 

7/5/2007 

4 

2 

0 

0 

2 

7/6/2007 

5 

0 

0 

4.5 

4.5 

7/6/2007 

6 

3 

4 

0 

4 

Totals 

8 

17 

9 

28 

Table  2.  Resource  setup  times  for  the  first  week  under  analysis. 


Note  that  the  value  for  “range  hours  impacted”  is  only  obvious  by  considering  a 
single  test  on  one  row  of  the  table.  The  range  hours  impacted  can  not  be  deduced  looking 
at  the  weekly  totals.  A  summary  of  the  weekly  totals  for  the  entire  12  weeks  is  presented 
in  Table  3. 

The  range  is  divided  up  into  several  sub-range  areas  to  allow  multiple  activities 
to  occur  on  the  range  at  the  same  time  when  conditions  pennit.  When  instrumentation  is 
set  up  on  several  sub-ranges  at  the  same  time  it  is  possible  to  see  large  numbers  for 
schedule  hours  impacted.  While  it  is  not  common  to  have  7 1  range  hours  impacted  in  a 
week,  as  they  were  on  the  week  ending  2 1  July,  it  does  happen. 

The  numbers  are  still  valid  data  for  analysis.  Each  impacted  range  hour 
represents  a  test  hour  that  cannot  be  conducted  in  an  area  of  the  range  because  setup 
activities  are  in  progress.  What  these  data  do  not  reflect  is  how  many  of  these  setup 
activities  are  conducted  on  overtime  either  early  in  the  morning  for  a  test  the  same  day,  or 
late  in  the  evening  for  an  event  the  next  day.  Even  with  this  overtime  consideration  the 
value  of  the  data  is  not  diminished.  Process  changes  that  result  in  less  setup  time  will  still 
be  a  relaxation  of  constraints  that  may  allow  improved  throughput. 
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Week  Ending 
(2007) 

KTM  Setup 
Hours 

Video  Setup 
Hours 

Laser  Setup 
Hours 

Range  Impact 
Hours 

July  7 

8 

17 

9 

28 

July  14 

14 

26 

2 

30 

July  21 

6 

18 

62 

71 

July  28 

19 

17 

20 

46 

August  4 

6 

12 

31 

35 

August  1 1 

14 

29 

36 

49 

August  18 

13 

23 

33 

46 

August  25 

5 

12 

18 

30 

September  1 

5 

7 

4 

13 

September  8 

8 

8 

13 

15 

September  15 

11 

3 

4 

15 

September  22 

9 

25 

20 

43 

Totals 

118 

196 

225 

424 

Table  3.  Data  representing  97  events  and  the  setup  time  required  by  each  resource. 


The  data  presented  in  Table  3  is  summarized  in  Figure  5.  Laser  Setup  Time  has 
the  greatest  impact  on  the  range  with  Video  and  KTM  setup  respectively  the  second  and 
third  greatest  contributors. 


Range  Hours  Impacted  by  Resource 


KTM  Setup  Hours  Video  Setup  Hours  Laser  Setup  Hours 

Resource 


Figure  5.  Range  hours  impacted  by  setup  by  resource  type. 
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Further  analysis  of  the  data  provides  insight  into  the  number  of  events  that  require 
various  combinations  of  resources.  These  data  are  presented  in  Figure  6  and  indicate  that 
more  events  require  only  laser  support  than  any  other  single  resource  or  combination  of 
resources.  Laser  resources  have  the  highest  impact  on  range  time  and  the  least 
correlation  with  other  resources,  making  a  reduction  in  laser  setup  time  the  most  likely  to 
have  a  positive  impact  on  range  time.  If  either  of  the  other  two  resource  setup  times  can 
be  reduced,  the  change  in  range  time  impacted  will  still  be  limited  by  the  other  resource. 


Events  by  Combination  of  Required  Resources 


Laser  Only  KTM  &  KTM,  Video,  KTM  Only  Video  Only  KTM  &  Video  & 
Video  &  Laser  Laser  Laser 


Figure  6.  Test  events  requiring  range  setup  by  resource  type. 


23 


The  number  of  events  requiring  combinations  of  resources  does  not  indicate 
which  resource  required  the  most  time.  The  data  was  further  analyzed  to  determine  the 
correlation  between  setup  times  of  each  resource.  Figure  7  is  the  result  of  that  analysis. 
There  is  consistency  in  the  ranking  of  laser  setup  taking  the  largest  amount  of  setup  time, 
but  this  analysis  shows  that  video  takes  the  second  largest  amount  of  range  setup  time. 
The  results  of  this  analysis  were  not  intuitive  to  the  author  or  to  most  of  the  range 
employees  to  whom  it  was  presented.  Time  was  spent  to  verify  the  data  to  the 
satisfaction  of  range  management  before  the  next  step  was  taken.  Preconceived  notions 
about  how  to  improve  throughput  were  abandoned  and  data  were  allowed  to  lead  the 
way.  The  order  of  priority  for  reducing  range  time  impacted  by  setup  will  be  on  laser, 
video,  and  KTM. 


Overlap  in  Setup  Hours  by  Resource 


Laser  Only  KTM  &  KTM,  KTM  Only  Video  Only  KTM  &  Video  & 
Video  Video,  &  Laser  Laser 

Laser 

Resource  Combinations 


Figure  7.  Setup  hours  by  resource  showing  the  overlap  in  setup  time. 
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c. 


IDENTIFY  PROCESSES  ASSOCIATED  WITH  INEFFICIENT  USAGE 


The  above  analysis  is  used  to  focus  on  range  processes  that  contribute  to  the 
amount  of  time  spent  on  the  range  setting  up  laser,  video,  and  KTM  systems. 

1.  Inefficient  Range  Usage  Resulting  from  the  Requirements  Definition 
Process 

Some  inefficiency  in  range  usage  stems  from  the  requirements  definition  process. 
This  inefficiency  is  caused  by  requirements  for  higher-precision  instrumentation  than  are 
actually  needed.  Interviews  show  that  not  all  test  managers  emphasized  the  need  for 
identifying  the  correct  required  precision  as  they  interfaced  with  customers.  Some  were 
satisfied  with  allowing  the  customer  to  use  a  perceived  range  precision  standard  as  the 
default  value  in  the  requirements  documentation.  This  practice  leaves  resource  providers 
few  options  other  than  designing  instrumentation  schemes  optimized  for  each  individual 
test  event  with  only  secondary  consideration  for  test  setups  that  will  work  for  multiple 
sequential  tests.  The  impact  of  this  practice  is  that  KTM  and  scoring  video  may  be 
required  with  the  associated  setup  time,  when  another  TSPI  solution  with  no  setup 
penalty,  such  as  GPS  or  radar,  could  have  been  used.  Even  if  KTMs  are  required,  an 
over-statement  of  TSPI  requirements  drive  the  placement  of  KTM  systems  closer  to  the 
target  area  and  increase  the  chance  the  systems  will  need  to  be  moved  for  the  next  test. 

2.  Inefficient  Range  Usage  Resulting  from  the  Scheduling  Process 

Scheduling  is  performed  with  a  view  that  looks  primarily  at  test  events  requesting 
test  conduct  the  following  week.  Relative  priority  plays  a  role  in  getting  the  requested 
time  slot.  These  two  practices  reduce  the  opportunity  to  utilize  a  single  target  location 
and  instrumentation  setup  for  multiple  test  events  that  might  be  scheduled  sequentially, 
thus  reducing  the  setup  time  between  events.  Once  again,  self-imposed  constraints  limit 
a  more  efficient  solution. 


25 


3. 


Inefficient  Range  Usage  Resulting  from  the  Instrumentation  Design 
Process 


Much  of  the  setup  time  associated  with  preparing  laser  and  video  systems  is  the 
time  required  to  ready  the  systems  for  transportation  and  then  to  move  them  from  the  last 
deployed  location  to  the  next  test  location,  set  them  up,  and  calibrate  them.  The  ranges 
cover  over  1700  square  miles,  and  thus  transportation  time  on  rough  dirt  roads  for 
sensitive  equipment  can  be  significant.  Data  on  KTM  setup  do  not  indicate  as  large  a 
time  requirement  for  transportation,  but  KTM  setup  time  would  benefit  if  transportation 
time  is  reduced.  Transportation  time  can  be  reduced  if  the  need  to  move  from  one 
location  to  another  between  tests  is  reduced. 

Laser,  video,  and  KTM  system  deployment  locations  are  driven  by  the  target 
location.  An  area  of  the  range  with  many  targets  is  Airport  Lake,  shown  in  Figure  8.  The 
range  has  hundreds  of  target  locations.  Unique  locations  and  the  use  of  very  specific 
targets  are  critical  requirements  for  some  programs.  Other  programs  have  greater 
flexibility  in  making  target  type  and  location  decisions.  Using  a  larger  number  of  target 
locations  increases  the  likelihood  that  instrumentation  setup  for  a  given  test  will  require 
equipment  relocation  and  calibration  time.  Both  targets  and  target  sites  must  be  rebuilt 
after  some  tests.  The  use  of  too  few  targets  sites  would  also  be  a  constraint  on 
throughput.  The  selection  of  a  set  of  optimal  targets  and  locations  is  a  good  topic  for 
another  study. 
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Figure  8.  Airport  Lake  supports  the  highest  concentration  of  weapons  impact  targets 

on  the  ranges. 

The  instrumentation  design  process  has  been  hampered  by  higher-precision 
requirements  than  needed,  as  discussed  previously.  Analysis  of  the  setup  process  and 
interviews  with  team  members  illuminated  additional  impacts.  The  instrumentation 
design  process  is  critical.  When  considering  placement  of  up  to  12  mobile  KTM  systems, 
at  over  200  surveyed  locations  on  range  there  are  6x10  combinations  to  choose  from. 
An  exclusion  area  is  an  area  where  people  are  not  allowed  during  a  specific  test  due  to 
the  potential  hazards  associated  with  the  test.  Some  KTM  sites  are  eliminated  from 
consideration  because  they  are  within  the  exclusion  area.  Consideration  for  placement 
must  include  good  geometry  for  data  processing,  as  well  as  sun  angle  for  the  predicted 
test  time,  air  quality  factors  that  can  degrade  image  quality  as  a  function  of  distance,  and 
geographical  features  that  may  block  the  item  from  the  cameras  during  portions  of  the 
test. 

Instrumentation  design  is  performed  by  key  individuals  with  years  of  experience. 
Using  experience  and  intuition,  setups  are  designed  that  meet  most  customers’ 
requirements.  The  number  of  key  individuals  capable  of  test  design  is  much  smaller  now 
than  in  the  past  due  to  downsizing  and  natural  attrition.  Due  to  challenges  with  site 
placement,  sun  angle,  difficult  tracking  conditions,  and  operator  proficiency,  risk  is 
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sometimes  reduced  by  deploying  more  systems  for  a  test  than  the  minimum  required. 
This  practice  may  result  in  the  inability  to  schedule  concurrent  tests  that  would  otherwise 
be  compatible. 

D.  DETERMINE  PROCESS  CHANGES  TO  IMPROVE  EFFICIENCY  AND 

IDENTIFY  IMPACTS  ON  OTHER  PROCESSES  AND  RESOURCE 

PROVIDERS 

1.  Proposed  Changes  to  the  Requirements  Definition  Process 

Meaningful  changes  to  the  requirements  definition  process  will  require  changes  in 
the  interface  between  the  test  manager  and  the  customers.  During  this  process  the 
flexibility  in  test  programs  must  be  identified  and  used  to  plan  test  events  with  solutions 
that  contribute  to  range  efficiency.  Specific  areas  of  flexibility  that  will  contribute  to 
improved  range  efficiency  include: 

•  Test  date  and  time 

•  Target  type  and  location 

•  Data  precision  for  TSPI 

One  challenge  will  be  customers  who  have  not  perfonned  the  analysis  within 
their  program  to  know  the  required  precision.  Customers  need  to  be  motivated  to  allow 
the  range  to  take  advantage  of  flexibility.  Benefits  for  the  customer  include  a  higher 
likelihood  of  getting  on  the  range  schedule,  and  the  potential  for  reduced  cost  associated 
with  actual  vs.  inflated  precision  requirements. 

To  meet  these  objectives,  specific  process  changes  to  the  requirements  definition 
process  are  recommended  as  follows. 

Test  Management  Processes  Handbook 

•  Chapter  2,  Initial  Requirements  and  Feasibility,  Paragraph  2.4  - 
Add  the  following  after  the  first  sentence,  “A  correctly  designed 
event  plan  is  less  restrictive  and  easier  to  schedule  than  one  with 
more  stringent  requirements.  Test  managers  are  to  get  TSPI 
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precision  requirements  from  the  customers  and  are  not  to  select 
this  value  for  the  customer  or  suggest  a  value.  Test  managers  will 
confer  with  customers  who  present  precision  requirements  that 
seem  excessive  for  the  specific  test  to  ensure  the  value  is 
technically  driven  by  key  performance  parameters  of  the  program. 
The  test  manager  will  inform  customers  of  the  added  cost  of 
increasing  TSPI  precision  requirements.  Test  managers  should 
encourage  customers  to  select  commonly  used  target  locations.” 

2.  Proposed  Changes  to  the  Scheduling  Process 

The  one -week  view  for  scheduling  is  a  self-imposed  constraint  that  has  many 
benefits,  such  as  keeping  the  process  simple,  being  able  to  ensure  that  the  highest  priority 
programs  each  week  get  test  time,  and  addressing  emerging  requirements  in  the  dynamic 
environment  of  Test  and  Evaluation.  These  benefits  were  considered  as  proposed 
changes  were  developed  in  order  to  try  and  keep  as  many  of  the  benefits  in  place  under 
the  modified  process  as  possible.  The  relative  priority  among  customers  remains  the 
same. 

The  goal  for  the  proposed  changes  is  once  again  based  on  a  relaxation  of 
constraints  to  allow  an  improved  solution.  The  relaxation  is  in  two  areas.  The  first 
relaxation  is  in  considering  a  longer  period  of  time  in  which  to  fit  the  requested  tests. 
The  second  relaxation  is  in  considering  a  larger  number  of  test  events  to  increase  the 
likelihood  of  identifying  events  that  can  be  conducted  consecutively  in  the  same  location. 
Logically,  the  larger  number  of  tests  considered,  the  more  that  will  be  found  to  share 
common  instrumentation  requirements.  Practically  the  solution  is  limited  to  the  number 
of  programs  sharing  similar  instrumentation  needs  that  are  ready  to  test  and  flexible 
enough  to  take  advantage  of  the  common  setup. 

It  only  takes  a  few  successes  in  matching  instrumentation  needs  and  scheduled 
times  and  locations  to  make  a  difference  big  enough  to  make  range  time  available  for 
another  event.  Success  of  procedure  changes  in  scheduling  depends  on  test  managers  and 
customers  as  much  as  on  the  scheduling  office.  Test  managers  need  to  encourage 
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customers  to  be  prepared  to  take  advantage  of  common  instrumentation  setups. 
Customers  who  have  their  event  plans  approved  and  some  flexibility  in  schedule  dates 
and  times  will  have  increased  opportunity  to  make  the  schedule.  This  may  be 
challenging  for  customers  who  rely  on  wings  for  aircraft  assignments  that  are  not 
determined  far  enough  in  advance.  Additional  coordination  between  the  range,  the 
customer,  and  the  air  wing  could  yield  solutions  to  this  challenge. 

The  information  in  the  Test  Management  Process  Handbook,  Chapter  6, 
Scheduling,  is  not  divided  into  paragraphs  or  sections,  and  so  it  is  difficult  to  propose 
changes  based  on  paragraphs.  The  entire  section  is  reproduced  below.  Paragraph  breaks 
have  been  added  and  the  paragraphs  numbered.  Suggested  deletions  are  shown  in  strike 
through  text,  and  additions  are  italicized.  The  proposed  changes  to  the  scheduling 
process  are  as  follows: 

“6.1  Responsibility 

All  events  conducted  on  the  Land  Ranges  are  scheduled  through  the  Land  Range  Test 
Management  Branch  Scheduling  offices. 

6.2  Long  Range  Forecasting 

Once  test  management  has  determined  that  the  initial  requirements  and  feasibility  of  an 
event  are  acceptable,  the  test  manager  checks  the  scheduling  calendar  for  availability  and 
initiates  the  scheduling  process  by  putting  the  event  in  the  forecasting  module  of 
submitting  a  request  for  an  Event  schedule  time  through  the  Test  Resource  Management 
System  (TRMS).  This  initial  request  contains  the  event  title  with  proposed  dates  and 
range  times.  Early  submittal  of  these  requests  gives  insight  to  test  managers  and 
supervisors  as  to  future  events  and  allows  them  to  balance  the  workload  for  most  efficient 
use  of  the  Range.  It  also  allows  for  de-conflicting  events  that  have  long  lead  times  in 
preparation  and  planning. 

6.3  Bringing  the  Event  Plan  to  the  Stage  of  Pending  Schedule  Approval 

The  test  manager  updates  the  Event  schedule  request  as  more  information  is  collected 
during  the  Test  Planning  process.  An  approved  test  plan  and  an  Event  Plan  with  all 
scenario  approvals  are  required  for  the  request  to  be  considered  for  scheduling.  As  the 
event  plan  is  populated  with  the  required  data  and  gains  the  needed  approvals  it  should 
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be  moved  from  the  forecasting  module  of  TRMS  to  the  scheduling  module,  the  following 
week’s  schedule.  Event  schedule  requests  are  displayed  as  “Pending  Scheduling 
Approval”  on  the  TRMS  Range  Schedule  as  soon  as  they  are  submitted  by  the  test 
manager.  The  deadline  for  submitting  requests  to  the  Scheduling  office  is  1000  hours 
each  Wednesday. 

6.4  Consideration  of  Long-Range  Scheduling 

At  the  test  managers’  biweekly  long  range  forecasting  meeting,  test  managers  will 
present  events  they  believe  are  appropriate  for  consideration  for  advance  scheduling. 
Considerations  for  advance  scheduling  include  program  priority,  intensive 
instrumentation  resources,  participation  and  coordination  of  non-local  assets,  maturity 
of  the  event  plan  with  details  and  approvals,  likelihood  of  completing  all  outstanding 
required  approvals,  confidence  in  the  customers  ’  ability  to  be  prepared  for  the  proposed 
test  period,  and  readiness  of  required  range  assets  to  meet  the  suggested  test  period. 
After  a  discussion  of  the  event  with  the  other  test  managers  the  branch  head  will  decide  if 
the  event  will  be  placed  on  the  schedule.  If  the  decision  is  to  schedule  the  event  then  the 
test  manager  moves  it  from  forecasting  to  scheduling  in  TRMS.  All  test  managers  then 
look  for  other  events  requiring  similar  instrumentation  capabilities  that  can  be  conducted 
in  the  same  location  that  could  take  time  slots  prior  to,  or  after  the  scheduled  event. 
These  additional  events  are  then  considered  for  placement  on  the  schedule.  Due  to 
considerations  for  range  efficiency,  these  programs  will  have  looser  requirements  for 
priority  than  did  the  originally  scheduled  event. 

6.5  The  Scheduling  Process 

The  Scheduling  office  uses  the  Event  schedule  requests,  events  previously  scheduled 
through  the  long-range  scheduling  process,  and  aircraft  schedules  to  prepare  the  daily 
schedules  for  the  following  week  (the  aircraft  schedules  should  already  have  resolved 
frequency  conflicts  between  the  aircraft  and  onboard  weapons  or  test  articles.  The 
Scheduler  assigns  proposed  range  times  through  prioritization,  resource  availability  and 
the  supervisor’s  discretion.  The  Scheduling  office  also  confirms  proposed  range  times  for 
events  that  will  use  other  Ranges’  airspace  or  resources.  Later  on  Wednesday  afternoon,  a 
review  of  the  daily  schedules  for  the  following  week  is  held  in  the  Range  Control  Center. 
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Test  managers,  a  test  management  supervisor  and  representatives  from  each  Range 
resource  attend  the  review,  being  attentive  to  resource  conflicts  and  clarification  or 
correction  of  event  support  requirements. 

6. 6  The  Final  Schedule 

After  the  meeting,  but  no  later  than  0900  Thursday  morning,  the  Scheduling  office  makes 
any  required  schedule  adjustments,  deletes  the  requests  from  TRMS  that  did  not  make  the 
schedule,  and  changes  the  remaining  events  from  “Pending  Scheduling  Approval”  to 
“Scheduled  Pending  Conduct  Approval,”  “Scheduled  Ready  to  Conduct,”  “Standby,”  or 
“Backup,”  as  appropriate.  The  Ready  to  Conduct  decision  is  made  by  the  Land  Range 
Test  Management  and  Test  Operations  supervisors  no  later  than  two  days  prior  to  the 
event.  Using  the  Integrated  Frequency  Deconfliction  System  (IFDS),  the  Frequency 
Management  Office  begins  identifying  and  resolving  frequency  conflicts  after  the  Final 
Schedule  is  published.  Changes  to  the  schedule  including  notification  of  the  test  manager 
are  the  responsibility  of  the  Schedules  office.  If  an  event  does  not  occur  due  to 
unforeseen  factors  (such  as  foul  weather  or  mechanical  failure),  the  test  manager  applies 
the  Test  Management  Cancellation  policy  for  an  equitable  allocation  of  costs  between  the 
customer  and  the  Range  Department.” 

3.  Proposed  Changes  to  the  Instrumentation  Design  Process 

In  order  for  the  relaxation  of  requirements  for  TSPI  precision  to  have  an  impact 
on  reducing  restricted  range  times,  the  flow-down  of  relaxed  requirements  must  translate 
to  an  improved  instrumentation  solution  (i.e.,  one  requiring  less  overall  setup  time).  If 
the  relaxation  allows  GPS  or  radar  systems  to  meet  the  test  objectives  in  place  of  KTM 
systems,  then  progress  has  been  made. 

There  is  potential  for  improvement  beyond  this  step  but  significant  cultural  norms 
make  such  advances  difficult  to  implement.  The  Instrumentation  Division,  like  other 
range  support  groups,  is  dedicated  to  quality  service  and  takes  pride  in  a  job  well  done. 
Getting  the  absolute  best  results  on  every  test  is  ingrained  in  the  China  Lake  culture  and 
in  the  individuals  who  work  here.  As  with  many  systems  there  are  competing 
parameters.  The  closer  to  the  test  item  the  KTM  is  placed,  the  greater  the  TSPI  precision 
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and  the  “wow”  value  from  the  product,  but  the  less  likely  the  placement  can  be  used  on 
the  next  event.  The  current  culture  is  dominated  by  the  desire  to  get  the  instrumentation 
systems  as  close  to  the  event  as  possible  in  order  to  obtain  the  best  possible  product. 

E.  APPLICATION  OF  PROCESS  CHANGES  ON  SELECTED  TEST  MONTH 

TO  IDENTIFY  IMPACT  OF  PROPOSED  CHANGES 

In  this  section  the  proposed  process  changes  are  applied  to  the  selected  baseline 
timeframe  and  the  scheduling  process  is  repeated.  The  results  of  this  scheduling  are 
compared  with  the  results  of  the  original  scheduling  to  provide  the  MOE. 

1.  Changed  Process  Simulation 

Scheduling  requires  a  large  amount  of  real-time  interaction  between  schedulers, 
test  managers,  customers,  and  resource  providers.  There  are  two  full-time  schedulers 
who  work  to  find  an  efficient  allocation  of  resources.  Scheduling  conflicts  are  avoided 
when  programs  are  flexible  in  one  or  more  of  three  areas.  First  is  the  ability  to  test  at  a 
different  time  on  the  same  day.  The  second  is  the  ability  to  move  a  test  to  another  day. 
The  third  is  the  ability  to  utilize  a  different  range  location  for  the  test. 

It  is  not  possible  to  re-create  the  exact  conditions  that  existed  a  year  in  the  past, 
nor  is  it  possible  to  allocate  weeks  of  time  from  the  vast  cast  of  characters  that 
contributed  to  the  original  scheduling  process.  The  scheduling  office  personnel 
performed  the  actual  scheduling  using  the  new  processes  in  a  manual  simulation. 
The  same  paper  schedule  requests  used  in  the  original  scheduling  from  baseline  dates 
were  used  as  the  input.  Simulation  of  scheduling  activities  for  a  period  of  time  in  the  past 
presented  a  few  challenges.  Simulation  is  only  possible  when  the  flexibility  of  the 
various  customers  can  be  considered.  To  allow  the  considerations  of  flexibility  within 
the  simulation,  test  managers  familiar  with  the  programs  provided  ratings  of  flexibility 
for  their  customers  who  requested  time  on  the  schedule  in  the  months  of  July  to 
September  2007.  Rating  values  and  the  applicable  rules  are  shown  in  Table  4.  Only  one 
in  four  events  with  moderately  flexible  ratings  were  allowed  to  flex  for  scheduling 
purposes.  These  tests  allowed  to  flex  were  selected  at  random  prior  to  the  simulation. 


33 


Area  of  flexibility 

Rating 

Rule  for  Simulation 

Time  of  Day 

1  -  Very  Flexible 

Can  move  the  event  to  another  time  on  the  same 
day  within  the  normal  constraints  of  scheduling 

2  -  Moderately 
Flexible 

One  in  four  events  with  this  rating  can  be  moved 
within  the  normal  constraints  of  scheduling 

3  -  Not  Flexible 

The  time  for  this  event  is  a  hard  requirement 

Day  of  the  Week 

1  -  Very  Flexible 

Can  move  the  event  to  another  day  of  the  same 
week  within  the  normal  constraints  of 
scheduling 

2  -  Moderately 
Flexible 

One  in  four  events  with  this  rating  can  be  moved 
within  the  normal  constraints  of  scheduling 

3  -  Not  Flexible 

The  day  for  this  event  is  a  hard  requirement 

Location  of  the  Test 

1  -  Very  Flexible 

Can  move  the  event  to  another  suitable  location 
on  range 

2  -  Moderately 
Flexible 

One  in  four  events  with  this  rating  can  be  moved 
within  the  normal  constraints  of  scheduling 

3  -  Not  Flexible 

The  location  for  this  event  is  a  hard  requirement 

Table  4.  Customer  flexibility  rating  assigned  by  test  managers  and  the  resulting  rule  to  be 

used  in  the  simulation. 

2.  Results  of  Simulation 

The  18  days  involved  in  the  rescheduling  simulation  included  11  work  days. 
China  Lake  is  on  a  schedule  that  provides  every  other  Friday  as  a  day  off.  August  13  was 
a  safety  stand-down  day.  As  required,  the  scheduling  under  the  proposed  procedures 
ensured  that  all  previously  scheduled  tests  were  placed  on  the  schedule.  The  daily  effect 
of  the  process  is  shown  in  Figure  9.  There  was  a  compressive  effect  in  the  new  process 
that  was  anticipated.  This  compression  better  utilized  days  early  in  the  week, 
creating  larger  blocks  of  available  time  at  the  end.  Larger  blocks  of  free  time  are 
much  more  useful  in  scheduling  additional  tests  than  are  smaller  time  periods.  The 
application  of  process  changes  resulted  in  eight  additional  free  range  hours  that  could  be 
used  for  testing.  The  process  also  reduced  the  need  for  overtime  flights  in  the  same 
period  by  four  hours. 
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Blocks  of  schedule  time  per  day 


Scheduled  Range  Events 
Range  Time  Available 
Range  Closure  (Safety  Stand-down) 
Overtime  Range  Events 


Figure  9.  Impact  of  the  new  processes  on  the  range  schedule. 

The  new  process  utilizes  unused  time  slots  earlier  in  the  week  and  frees  up  larger 

blocks  of  time  for  later. 

The  eight  hours  reclaimed  by  the  new  process  represent  nearly  9%  of  the 
schedulable  time.  This  time  can  be  used  for  additional  testing.  The  exact  amount  of 
testing  that  could  be  fit  into  the  reclaimed  hours  depends  on  the  details  of  the  tests.  One 
large  test  event  with  complex  instrumentation  requirements  could  take  nearly  the  entire 
time,  or  many  less  intensive  tests  could  share  the  airspace  and  other  resources.  Based  on 
the  average  time  for  tests  in  the  baseline  time  period,  an  additional  six  test  events  could 
be  conducted  in  the  eight  hours  provided. 
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In  the  event  that  no  additional  tests  are  requested,  training,  maintenance, 
calibrations,  and  other  needed  range  activities  can  be  accomplished.  The  larger  blocks  of 
available  time  are  more  useful  for  these  activities  than  the  equivalent  amount  of  time  in 
smaller  blocks. 


3.  Discussion  of  Results 

This  manual  simulation  is  susceptible  to  errors  as  are  all  simulations.  Not  all 
proposed  process  changes  could  be  sufficiently  simulated.  The  requirements  definition 
process  changes  must  be  applied  to  actual  customers  and  would  involve  lengthy  and 
detailed  analyses  involving  many  customers  and  their  supporting  analysts  in  the  initial 
planning  phase.  If  the  recommendations  in  this  thesis  are  implemented,  the  weekly 
scheduling  task  would  begin  with  some  tests  firmly  scheduled  through  the  advance 
scheduling  process.  To  prevent  an  overly  optimistic  outcome,  the  rules  of  the  simulation 
limited  the  maximum  flexibility  of  a  test  to  another  time  slot  in  the  same  week.  This 
restriction  will  not  exist  when  the  proposed  processes  are  implemented. 

There  were  known  weaknesses  in  the  simulation.  Resource  setup  times  were  not 
recorded  from  the  actual  events,  but  estimated  by  the  providers  after  the  fact.  A 
simulation  of  this  advanced  scheduling  on  the  first  week  was  not  performed  due  to 
difficulty  in  simulating  the  process  change  in  the  forecasting  meeting  a  year  after  the  fact. 
The  flexibility  ratings  assigned  by  the  test  managers  were  estimations.  In  retrospect,  it  is 
possible  that  the  simulation  rules  for  tests  that  were  “very  flexible”  should  have  included 
a  restriction  as  did  those  for  “moderately  flexible.”  If  the  simulation  were  run  again,  the 
suggestion  would  be  to  only  allow  half  of  the  “very  flexible”  tests  to  be  moved. 

Additional  insights  and  a  more  accurate  representation  of  the  impacts  of  the 
process  changes  could  be  detennined  with  a  simulation  that  considered  a  longer  period  of 
time.  A  longer  simulation  would  show  the  impact  of  the  sliding  window  effect  after  a 
few  weeks.  The  time  demands  on  the  scheduling  office  prevent  repeating  or  extending 
the  simulation.  Using  persons  other  than  the  schedulers  to  perform  further  simulation 
would  lack  an  element  of  realism  needed  to  give  credibility  to  the  results. 
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Restrictions  on  flexibility  would  likely  reduce  the  benefit,  while  implementing 
advanced  scheduling  and  a  longer  simulation  would  increase  the  benefit.  The  potential 
for  skewing  the  simulation  with  these  inconsistencies  was  discussed  with  the  participants 
before  and  after  the  simulation.  All  participants  agreed  the  simulation  had  weaknesses, 
but  that  the  potential  for  improvement  shown  in  the  results  was  valid. 

F.  CHAPTER  SUMMARY 

The  after-the-fact  scheduling  simulation  was  not  a  perfect  representation  of  the 
process  but  rules  were  applied  to  make  it  a  reasonable  representation.  The  resulting  data 
show  strong  potential  for  improved  throughput  if  the  proposed  process  changes  are 
implemented. 
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V.  CONCLUSIONS 


A.  KEY  POINTS  AND  RECOMMENDATIONS 

Application  of  the  proposed  process  changes  to  historical  data  gives  a  strong 
indication  of  the  potential  for  improved  throughput.  The  process  changes  proposed  in 
this  paper  are  recommended  for  implementation.  It  will  take  time  for  the  proposed 
processes  changes  to  take  hold  with  the  range  employees  and  with  customers. 
Management  will  need  to  follow  through  to  ensure  the  improved  processes  are 
implemented  and  followed. 

B.  AREAS  TO  CONDUCT  FURTHER  RESEARCH 

The  process  of  collecting  the  data,  interviewing  employees,  and  analyzing 
processes  illuminated  other  areas  that  would  potentially  benefit  from  further  study.  The 
Range  Department  currently  supports  a  strong  continuous  process  improvement  (CPI) 
program  with  personnel  trained  in  the  use  of  many  tools.  It  is  hoped  that  some  of  the 
suggestions  below  could  become  efforts  of  the  CPI  program. 

1.  Selection  of  Best  Target  Locations  and  Optimum  Number  of  Targets 

Target  locations  on  the  ranges  have  been  designated  by  testers  over  the  past  65 
years.  Many  target  locations  have  low  value  because  of  the  poor  line  of  sight  to  current 
instrumentation  sites,  or  are  located  a  long  distance  from  good  roads.  Many  of  these  poor 
locations  continue  to  be  used.  There  are  hundreds  of  surveyed  target  locations  on  the 
range.  Further  study  is  recommended  to: 

•  Determine  the  number  of  target  locations  needed  to  efficiently  meet  the 
needs  of  the  range  customers. 

•  Identify  the  best  target  locations  based  on  customer  needs  and  on  range 
supportability. 

•  Provide  documentation  of  the  resulting  targets  and  locations  for  use  by  test 
management  in  communicating  target  options  to  the  customers. 
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2.  Improve  Rigor  in  Test  Design  Including  Better  Access  to  Common 
Support  Tools 

Interviews  highlighted  the  fact  that  more  scientific  rigor  should  be  applied  to  test 
design.  It  was  also  noted  that  some  range  personnel  had  not  been  aware  of  tools  that 
could  assist  them  in  the  design  of  instrumentation  setup.  An  example  of  such  an  instance 
is  the  selection  of  KTM  placement  sites  without  the  aid  of  mapping  tools  such  as  shadow 
graphs.  An  assessment  of  the  test  design  processes  on  the  ranges  would  be  a  large 
undertaking  but  could  provide  significant  insights  into  areas  that  need  improvement. 

3.  Culture  Change  to  Allow  More  Efficient  Instrumentation  Setup 

Significant  benefits  from  relaxed  precision  requirements  will  only  be  possible 
when  the  desire  of  instrumentation  personnel  is  to  find  the  maximum  distance  from  target 
that  their  systems  can  be  located  and  still  meet  customer  requirements.  In  such  an 
environment  one  could  realize  additional  benefits  through  an  increase  in  the  number  of 
instrumentation  setups  that  meet  multiple  test  event  requirements.  The  previous 
recommendation  will  need  to  be  fully  implemented  before  this  one  is  worth  addressing. 

4.  Development  of  Permanently  Instrumented  Test  Sites 

The  impact  of  instrumentation  resources  required  for  setup  on  Baker  Range  (a 
sub-division  of  the  Land  Range)  was  noticeable  by  its  lack  of  impact  on  range  setup  time. 
This  is  because  Baker  Range  has  a  permanently  installed  camera  scoring  system.  If  the 
results  of  the  target  location  study  suggested  in  part  1  show  a  location  or  locations  that 
could  be  used  by  many  tests,  then  the  sites  should  also  be  studied  for  the  feasibility  and 
benefits  of  permanent  instrumentation. 
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